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Abstract. Using a general result of Lusztig, we give explicit formulas for 
the dimensions of if^-invariants in irreducible representations of G F ', when 
G = GL n , F : G — + G is a Frobenius map, and K is an F-stable subgroup 
l/~) , of finite index in G s for some involution 9 : G — » G commuting with F. The 

Cn ' proofs use some combinatorial facts about characters of symmetric groups. 



i-Q ' Introduction 

Let G be a connected reductive group defined over a finite field ¥ q of odd char- 
acteristic. Let F be the Frobenius morphism on G, whose fixed-point subgroup 
G F = G(¥ q ) is finite. Let 9 : G — > G be an involution of algebraic groups com- 

£N| ' muting with F, and K an F-stable subgroup of the fixed-point subgroup G which 

contains (G e )°. The homogeneous variety G/K is a symmetric space, and the set 
of cosets G F J ' K F might reasonably be called a finite symmetric space. 

The classification of irreducible representations of G F (in characteristic 0) was 

[^ ■ completed by Lusztig in the mid-1980s (see || for a general statement, and || and 

|9l for more details). A roughly analogous problem for symmetric spaces is that of 
determining the dimension of the iC F -invariants in each irreducible representation 
of G F ; equivalently, calculating the multiplicities (x, Ind K f (1)) for every irreducible 

,-Ch character \ of G F . A large first step towards solving this problem was Lusztig's 

calculation, in [pL0|| , of (tr(-,i?^),Ind isr i?(l)} for every Deligne-Lusztig virtual repre- 
sentation R^ of G F . In [O , Lusztig proceeded to solve the problem completely in 
the case when G F = (G') F , K F — (G') F , and G' has connected centre. 

This paper is devoted to the solution of this problem when G is a general linear 
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group (with either split or non-split F, so that G is either GL n (¥ q ) or [/„(F 9 2)) 
S . and 8 is arbitrary. (The solution for the case G F = GL n (¥ q ), K F = Sp n (¥ q ) 

was found by Bannai, Kawanaka, and Song in [jj], §4].) For such 67, the functions 
tr(-,i?j.) form a basis of the class functions, and the transition matrix from this 
basis to that of the irreducible characters is known. So Lusztig's result gives a 
formula for (x,Ind^-F(l)). All that remains is to manipulate this formula until it 
is manifestly a nonnegative integer, a straightforward (though not entirely easy) 
matter. Two justifications for presenting it in detail are the potential interest of 
the answers, and the pleasantness of the symmetric group combinatorics involved. 
In §1 we recall Lusztig's formula and the character theory of the finite gen- 
eral linear and unitary groups as well as introducing some vital notation. Then 
we traverse the various cases in §§2-4, which could be thought of as a theme 
and variations: the theme, or underlying pattern, is stated in its simplest form 
in §2.1 (the case already known from [Q), and successive subsections follow the 
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same pattern with progressively more elaborate alterations. In §2 the involu- 
tion is symplectic, so n must be even and the possible symmetric spaces are 
GL n (¥ q ) J Sp n (¥ q ) and U n (¥ q 2)/Sp n (¥ q ). In §3 the involution is inner, so the possi- 
ble symmetric spaces are GL n {¥ q )/(GL n + (¥ q ) x GL n - (¥ q )), GL n (¥ q )/GL n / 2 (¥ q 2), 
U n (¥ q 2)/(U n +(¥ q 2) x U n - (F g2 )), and U n {¥ q 2)/U n/2 {¥ qi ). In §4 the involution is or- 



thogonal, so G is not connected. However as noted in Lemma 4.0.1, it is enough to 
solve the problem when K = G e , in which case the possible symmetric spaces are 
GL n (¥ q )/O n (¥ q ) (n odd), GL n (¥ q )/0±(¥ q ) (n even), U n (¥ q2 ) / O n (¥ q ) (n odd), 
andC/„(F 9 2)/0±(F g ) (n even). 

The key combinatorial results we need along the way are all proved in §5. A 
reader interested only in these results could skip all of §1-4 except §1.2. 

To give some idea of how the formulas in §§2-4 connect with previously known 
results, we here extract the answers for unipotent irreducible characters. For both 
GL n (¥ q ) and U n (¥ q 2), these are parametrized by partitions of n, say p t-» \ p G G F . 
(In our convention x is the trivial character and x is the Steinberg character.) 
Recall that a signed tableau of shape /i is a signed Young diagram of shape p where 
signs alternate across rows, modulo permutations of rows of equal length. For this 
and all other combinatorial notation, see §1.2. We have: 



(neven) (xMndff"^ (1)) 



1 , if p is even 
Sp n (¥ q ) w/ - i q^ otherwise 



(for the general GL n (¥ q )/Sp n (¥ q ) case, see Theorem 2.1.1 ) 



, . , p A u n (¥ 2 ) ( 1 if pis even 

(neven) (x p ,1*d Sp J q) (l)) = [ Q> otherwige 



(for the general U n (¥ q 2)/Sp n (¥ q ) case, see Theorem |2. 2. l| ); 



/ p T ,GL„(¥ q ) fi\\_ the number of signed tableaux of shape // 

\X ) J JMGL B+ (F,,)xGi B _(F,)W/- and signature (n+,n-) 

(for the general GL n (¥ q )/(GL n +(¥ q ) x GL n -{¥ q )) case, see Theorem |3~i~l"l ); 



, , , „ t iGL„(¥„) , lN , the number of signed tableaux of shape p' 

m even (y p ,Ind^ r q ,l Al)) = , ,, jr.- n ■ 

v ' s (jL »/2( , ',2) x " stable under changing all signs 

1 , if p is even 
0, otherwise 



(for the general GL n (¥ q )/GL n / 2 (¥ q 2) case, see Theorem |3. 2. l|) ; 



the number of signed tableaux of shape p' 

(x P , Ind [/ + (iF 2 )xC7„_(F 2)^)) = and si g na ture (n + ,n~), stable under 

inverting all rows 
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(for the general U n (¥ q 2 )/{U n + (F g 2) x U n -(¥ q 2)) case, see Theorem 3.3.1); 



the number of signed tableaux of shape p' 

U n (V 2 ) 

(n even) (x p ,Ind (7 L v(l)) = stable under changing all signs 

and inverting all rows 

n( TO 2,(p') + 1), if2|m 2i +i(p'), Vt 

i 

0, otherwise 



(for the general U n (¥ q 2)/U n / 2 (¥gi) case, see Theorem |3. 4.1 



(n odd) (xMnd^^l)) = ilJ^W + X ) 
(for the general GL„(F g )/0„(F g ) case, see Theorem [4. l.l| ); 

1 -1— r 

l"o II ("^^ + 1 )1 ' if P' is even ' e 

Lo n^ w + ^j ' if p' is even ' e 



(neven) ^Ind^^l)} = <j 



- TT(toj(p) + 1), otherwise 



(for the general GL n (¥ q )/0^(¥ q ) case, see Theorem [4.2. l| ); 



(n odd) (x p ,Ind 0ri(r 5 g) (1)) = ^ 2 Y 

0, otherwise 



(for the general U n (¥ q 2)/O n (¥ q ) case, see Theorem |4. 3. l|) ; and 



even) (x^Ind^Jf^l)) = \ 



l-^Yl{rn2i(p) + l)], if p' is even, e = + 
L2 IX(" l 2i(jo) -+- 1)J, if p' is even, e = - 

2 liv w 1 " but p' not even 



0. 



otherwise 



(for the general U n (¥ q 2)/0^(¥ q ) case, see Theorem 4.4.1). 

Readers experienced in the theory of cells for the symmetric group will find these 
answers familiar. In fact, most of the above facts about unipotent characters can 
be obtained by a more direct method than the one used in this paper. For instance, 
suppose that G F = GL n (¥ q ). Then the unipotent irreducible characters are the 

constituents of Ind^ F (1) where B is an F-stable Borel subgroup. The Hecke algebra 
7i(G F , B F ) is the specialization at q of the abstract Hecke algebra H of S n ; let V p 
be the simple 7i-module indexed by p. 

It is trivial to show that the above multiplicity (x p ,l'nd K F(l)) equals the mul- 
tiplicity of (V p ) q in the H(G F , B F )-modvle C(B F \G F /K F ) of functions on G F 
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which are constant on the B F -K F double cosets. Assuming that K is connected 
and split over ¥ q , this module is the specialization at q of the 7Y-module M K de- 
fined in pa]. So (x p ,Ind^F(l)} is the multiplicity of V p in M K . In the current 
type- A case, this equals the number of cells of M K which afford the representation 
V p , for which there is a combinatorial formula. For example, our answer in the 
case of GL n (¥ q )/(GL n +(¥ q ) x GL n - (¥ q )) could be deduced from the results in B- 
When G and K are non-split, this argument must be refined to incorporate folding 
involutions, in the manner of M, §10]. 

In principle, such arguments apply to (x-,Ind KF {l)) whenever \- is a con- 
stituent of Ind^F(A), since the generalized Hecke algebras Endc-plnd]^ (A) have 
been completely described. But the requisite facts about cells for these Hecke al- 
gebras are somewhat diffuse in the literature, and usually quoted in the slightly 
different context of real Lie groups. I hope that the results of this paper, which 
are deduced from |1C[| in an independent way, will in fact shed further light on the 
theory of cells. 
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k" -invariants in irreducible representations of g" , when g = gl„ 5 

1. Review of Known Results 

In §1.1 we recall the theorem of Lusztig which underpins all our results, and in 
§1.3 the relevant parts of the character theory of the finite general linear and unitary 
groups in a convenient form. In §1.2 and §1.4, we introduce some combinatorial 
notation to be used throughout the paper. 

1.1. Lusztig's Formula. Let k be the algebraic closure of a finite field ¥ q of 
odd cardinality q. Let G be a connected reductive group over k defined over ¥ q , 
with Frobenius map F : G — > G. Let : G — ► G be an involution of algebraic 
groups commuting with F, and K an F-stable subgroup of the fixed-point subgroup 
G which contains (G e )°. Fix a prime I not dividing q. All representations and 
characters of finite groups in this paper will be over Q; . 

A pair (T, A) means an F-stable maximal torus T and a character A : T F — » Q; . 
We have a conjugation action of G F on the set of pairs. In |J, Deligne and Lusztig 
attached to each pair a virtual representation R^ of G , depending only on the 
G -orbit of (T, A) . (In general, "most" of the R^ are irreducible up to sign, and 
every irreducible representation occurs in some R^; when G = GL n the situation 
is even better, as we will see below.) The main result of [J10|| is a formula for 

-^ Yl tr(fc,i^) = (tr(.,i4),Ind^(l)}. 

After some trivial adjustments, it reads as follows. Define 

eT = {.feG\9(f- 1 Tf) = r 1 Tf}. 

Then T acts on Qt by left multiplication and K acts by right multiplication. If B is 
a Borcl subgroup containing T, the obvious map T\Qt/ K — > B\G/K is a bijection 
(see [|10| Proposition 1.3]), so T\&t/K is in bijection with the set of if-orbits on 
the flag variety. For any / £ 9£, define e T j : (T n fKf- 1 ) F -> {±1} by 

£T U\ _ (_ 1 )F g -rank(Z G ((Tn/K/- 1 )°))+F,-rank(Z° (t)nZ G ((Tn/K/- 1 )°)) i 

It follows from pOl Proposition 2.3] that e^j is a group homomorphism which 
factors through (T n fKf~ 1 ) F '/((T n fRf' 1 ) ) 1 ". Finally, define 

©T,A = {/ e ©T I ^(Tn/K/- 1 )^ = £ T,/}, 

a union of T F -K F double cosets. 

Theorem 1.1.1. (Lusztig, @ Theorem 3.3]) 

(tr(-,i?*),Ind^(l)) = ^ ( _ 1) V r a„k(T) + F,-rank(Z G ((Tn/^/- 1 )°))_ 

1.2. Combinatorial Notation. In general, our combinatorial notation always fol- 
lows |14| . For instance, /j,\- n means that /j, is a partition of n. The size of a partition 
/i is written |/i| and its length £([i); it has parts /ii, /i2, . . . , /^( M )- The transpose 
partition is [i! . We define 



^)=£(i-l)^=£(^). 



The multiplicity of i as a part of \x is written m,i(n). We say that /i is even if all 
its parts are even, or equivalently if 2|mj(//), Vi. 
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It will be useful to have, for any partition v, a concrete realization of the sym- 
metric group S\ u \ which includes a canonical element of cycle-type v. Let A(y) be 
a set indexing the parts of v\ usually, A(y) = {1, • • • ,£(v)}. Then the set 

n(i>) = {(j,*)|jeA(i/),tez/i/ i z} 

has \v\ elements, and we will write S\ v \ for the group of permutations of Cl(y) (a 
slight abuse of notation, since it is not canonically associated to \v\). Let w v be 
the permutation (j,i) i— ► (j,i + 1). By construction this has cycle-type v, and in 
fact we can identify A(v) with the set of cycles of w v . Write Z v for the centralizcr 
Zs\ u \ {w v ). The sign of w v is written e v , and the size of Z v is z v . 

For any w G Z", let w be the induced permutation of A(v). Note that Vw(j) = v j 
always. For w G Z v and j G A(v), there is a unique i(w,j) G Z/z/jZ such that 

w(j,i) = (w(j),i + i(w,,?)), Vz G Z/i/jZ. 

We will have much to do with the set Z" m of involutions in Z y . Also define Z^_- lnv , 
the set of fixed-point free involutions. Clearly w G Z v lies in Z;^ v if and only if w 
is an involution and 

i(w,w(j)) = -i{w,j), V? G A(v). 

Note that any w G i^ v divides the set A(y) of cycles of w v into three disjoint 
subsets A^(i/), A^,(f ), and A^(z/), according to whether w fixes the cycle pointwisc, 
fixes the cycle but not pointwise, or does not fix the cycle. Explicitly, 

Ai,M = {j G A(i/) I w(j) = j,i(w,j) = 0}, 
Aj» = {j G A(i/) | w(j) = j,i(w,j) = -vj}, 

Al(v) = {jeA(v)\w(j)^j}. 
We will write l\,(v), l 2 w {v), and t%,(v) for |Ai,(z/)|, |A^(z/)|, and |A^(i/)|, so that 

0")+^(") + ^(") =*(")• 

Note that ^(v) is always even, and to G Z^_ inv <=» ^i,(^) = 0. 

We will use the notations £(v)o and £(v)i for the number of even and odd parts 
respectively. For instance, e„ = (— 1)^")°. On occasion we will need to further 
analyse £(v) into £(z/)o mo d4 and ^(^) 2m od4- We will also combine these notations 
in the obvious way, e.g. £%(v)i means the number of odd cycles of w v moved by w, 
and l\,{y)\ = always. 

In §3 we will need to consider involutions with signed fixed points. Let (S\ v \ )±-inv 
be the set of pairs (w, e) where w G (5|^|)i nv and e : {(j, i) \ w(j, i) = (j, i)} — » {+, — } 
is a way of signing the fixed points of w. We define the signature of (w, e) to 
be (|e _1 (+)|, |e _1 (— )|). We declare that signatures are considered as elements of 
Z 2 /Z(l,l), e.g. (1,0) and (2,1) are the same. Let (-S , |^|)(p+.p-)-i nv be the set of 
(w, e) G (S l | jy |)±_i nv with signature (p + ,p~). Now define 

Z±_ inv = {(w,e) G (S'| iy |) ± -i n v \w G Z",eow v = e}. 



Note that 



Define ZY + -\_ inv similarly, and also 



Zl ■ - V 2^» 

weZ inv 

and also 
^*-inv = {( w . e ) e (S'| t/ |)±_inv|w ^",eo% = -e}. 
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Note that any (w,e) 6 Z"_ inv must have signature (0,0). Also 

Let T±{p) be the set of signed tableaux of shape \x. These are ways of labelling 
the boxes of the Young diagram of p, with a sign, in such a way that signs alternate 
across each row (so all signs in a row are determined by that of the last box), with 
the proviso that two labellings which differ by a permutation of rows of equal length 
are not distinguished. Hence 

\T ± (n)\=H(m l ( f i) + l). 

i 

For T G T±(p), the signature of T, again in Z 2 /Z(l, 1), is defined as 

(| {boxes of T signed +}|, | {boxes of T signed — }|), 

or equivalently 

(| {odd rows of T ending EH}|, |{odd rows of T ending B}|). 

Write 7( p + p - \ (/x) for the set of T e 7± (/x) with signature (p + ,p~). There are two 
important involutions on 7±(/x): ip which changes all signs, and ip which reverses 
all rows. Write T±(p) lf> etc. for the fixed-point sets. Note that any T in T±(/i) ip or 
7±(/i) v ' / ' must have signature (0,0). Also 



\T±{nf\ 



\r±{u) 



tpipi 



1, if 2 | mi(fi), Vi 
0, otherwise, 

Y\_{m 2i+ i(ti<) + 1), if 2 | m 2 i(p'), Mi 

i 

0, otherwise, and 

JJ(m 2 i(M) + 1): if 2|m 2 i+i(/x), Mi 

i 

0, otherwise. 



There are no similar formulas for \Tf p + ij-)(m)I or \T( P + .p-){p)^\■ 
We label the irreducible characters of 5|„| as {x p \ P ^ Ml as m @' ^■•^]' anc ^ 
write x1 f° r t ne value of x p at an element of cycle-type v, so that x^ u ^ is the trivial 
character and x p v — e »xt- 

1.3. Character Theory of GL n (¥ q ) and C/ n (F ? 2). For the remainder of the pa- 
per, we specialize the context of §1.1 drastically, to the case when G = GL n , for 
some positive integer n. More concretely, let V be a vector space over k of dimen- 
sion n and let G = GL(V). There are two kinds of F 9 -structures on G, split and 
non-split. A Frobenius map F : G — > G is split if it is induced by some Frobenius 
map Fy : V — * V, in the sense that 

F v (gv) = F(g)F v (v), MgeG,veV. 

Then G F = Autp (V Fv ) = GL n (¥ q ), the finite general linear group. If F is a non- 
split Frobenius map, there exists some outer involution 6' of G commuting with F, 
and for any such 9' , Q'F is a split Frobenius map. In this case G F = U n (¥ q 2), the 
finite unitary group. 
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As in []14[ Chapter IV], we will need to consider the system of maps F x c — > F x , 

for e | e' (the transpose of the norm map) , and its limit L = colim F x e . Let a denote 
the g-th power map on both fc x and L, so that (k*) a ° = F x „, L a ° = F x e for all 

e > 1. Write (•, •)" : (/c x ) cr xi° — * Q; for the canonical pairing. Let i denote 
the inverse map on fc x and L, and write a for 10, the (— g)-th power map. (Note 
that a 2 = a 2 .) We also have a canonical pairing (■, ■) a " : (k x ) a ° x L 17 " — + Qj (the 
same as (•, •} cr if e is even). 

We will fix some set of representatives for the orbits of the group (a) generated 
by a on L, and call it {a)\L. Similarly define (5)\L. For £ 6 (cr)\L, let m^ = |(er).£|, 
in other words the smallest e > 1 such that c e (£) = £. Let d,£ = (— 1,£) CT ) which 
equals 1 if L CT £ contains square roots of £, and —1 if it does not. Similarly define 
m£ and d^ using a instead of a. 

First consider the case when F : G — » G is a split Frobenius map. Let "P n 
be the set of collections of partitions (i = (p a )aEk x ) almost all zero, such that 
J2aek x l/^-ct | = n. Let V^ be the subset of V n consisting of all \i such that H a r a ) = \x a 
for all a. It is well known that there is a natural bijection between V^ and the set 
of conjugacy classes in G F . Dually, let V n be the set of collections of partitions 
y_ = (i/f)f eI ,, almost all zero, such that J2teL Wzl = n - Let P° be the subset of P n 
of all v_ such that v a (g\ = v% for all £. Note that for v_ s 7^, 

Ce(<r)\L 

For v, p e 7^, we write |j/| = |p| to mean that \v$\ = |p{| for all £. 

We can define a bijection between T 3 ^ and the set of G F -orbits of pairs (T, A) as 
in §1.1, so that if (T, A) is in the orbit corresponding to u: 

1. the eigenlines of T can be labelled 

{L (U ,i) I £ e (*>\A ! < J < «> * e Z/m c (i/ € ),-Z} 
so that under the resulting isomorphism 

t - n n fe x x-'-xfc^ 

Ce<cr)\L j = l mi ( UlL ) j factors 

-F|t corresponds to cyclic permutation of each group of factors fc x , composed 
with a; 

2. consequently, 

tf = n nc**)*" 1 ^ 

Ce<<r)\L i=i 

3. under this isomorphism, A corresponds to 

n ik^)^^ ,j . 

«e<<r)\L i=i 

For £ e V%, let 73„ = tr(-,i?£) for (T, A) in the corresponding G F -orbit. As 
proved by Lusztig in Q , these coincide with the basic characters defined by Green 
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in p|. (In particular, their values are computable, but this is not relevant here.) 
Green's main result on the character theory of GL n {¥ q ) states that for any p £ P°, 

x P := (_l)"+£, eMX ^l £ | J[ (z Vi )- l X p A B„ 

\a\=\p\ 

is an irreducible character of G , and all irreducible characters arise in this way 
for unique p€^, (See also @, Chapter IV] and §], Theorem 1.2.10]. Note that 

Macdonald's parameters in P° differ from those of pj by transposing all partitions; 
we are following the convention of EL) In words, the transition matrix between the 
basic characters and the irreducible characters is formed from the character table 
of various symmetric groups. Inverting this matrix, we have that for any y_ £ P%, 



(1.3.1) B !L =(-l) n+E ^< 



"5 




There is an obvious action of L a on P°, and in particular, for v £ P°, Q.v is 
well defined. For any -q e L CT , B niL and x v - are the result of multiplying B^ and 
X- by the one-dimensional character (det(-), r/)' 7 of G F . The unipotent irreducible 
characters referred to in the introduction are those x- f° r which p^ = unless 
£ = 1. (In the introduction we parametrized these by p — p±.) 

The case when F : G — > G is a non-split Frobenius map is less well known, but 
very similar, in fact mostly identical once a is replaced by a, m^ by mf, and so on. 
Define P% in the obvious way. Again, for any y_ g P° , 

For v,p£ 7-^, we write \u\ = \p\ to mean that \v^\ — \p$\ for all £. 

We can define a bijection between P° and the set of G F -orbits of pairs (T, A) as 
above, so that if (T, A) is in the orbit corresponding to v\ 

1. the eigenlines of T can be labelled 

{L(UA I £ e (a)\L, 1 < j < *(i/ c ), i G Z/m^^-Z} 
so that under the resulting isomorphism 

r - n n fc x x • • ■ x k x , , 

ie(5)\L 3 = 1 m € (f € )j factors 

.F|t corresponds to cyclic permutation of each group of factors fc x , composed 
with ex; 

2. consequently, 

T F = u U(k«r^ h ; 

te(a)\L j=i 
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3. under this isomorphism, A corresponds to 

n n <••«>*" 



«("«)i 



For 1/ £ £>£, let B„ = tr(-, i$) for (T, A) in the corresponding G F -orbit. The ex- 
tension of Green's result to the non-split case was proved by Lusztig and Srinivasan 
in 113, Theorem 3.2]: in our notation, for any p £ P°, 

X £ := (_i)rsi+£ €6<ffni *«»(Pi)+i*i j2 J yi (^r^ijBj, 

IkNIpj 

is an irreducible character of G F , and all irreducible characters arise in this way 
for unique p £ P%. Inverting, we see that for any v_ £ P%, 

(1.3.2) Bu= J2 (-l) f51+E «t»>\ i *« B ^ )+l « l I n X P 4]X^ 

IpI=IkI 

Again, the obvious action of L a on P% corresponds to multiplication by one- 
dimensional characters, and the unipotent irreducible characters of the introduction 
are those x- f° r which p^ — unless £ = 1. (In contrast to the case of GL n (¥ q ), 
not all the unipotent characters are constituents of Ind B F(l) for an instable Borel 
subgroup B.) 

1.4. Descriptions of Weyl Groups. Much of this paper deals with the special 
properties of Weyl groups in GL n , so the following ideas and notation will be 
crucial. Let T be any maximal torus of G = GL(V). If {L^ \u> £ 0} is some 
labelling of the eigenlines of T, we can identify the Weyl group W(T) with the 
group of permutations of fi. Let W(T)inv be the set of involutions in W(T), and 
V^(T)ff_i nv the set of fixed-point free involutions (note that this means fixed-point 
free as a permutation of fi, not as an automorphism of T). 

If T is 6>-stable, there is a special involution wf £ W(T)- lmr characterized as 
follows. If 9 is an inner involution, namely conjugation by s £ G, then 

s(L u ) = L w t^, Vw £ Q, and 9\t = wj '. 

If 9 is an outer involution, namely adjoint inverse with respect to a nondegenerate 
symplectic or symmetric form on V, then 

^u = (J) Lu', Vw £ fl, and 9\ T = w[ o i. 

For any T, if / £ 9t, then f~ x Tf is a 0-stable maximal torus, and we obtain 

wj = Ad(/- 1 ) o wf lTf o Ad(/) £ W(T) inv . 

When T is fixed, we will write Wf instead of wj . Clearly Wf depends only on the 
double coset TfK; the resulting map T\&t/K — > W(T) inv will be crucial in our 
combinatorial rewritings of Lusztig's theorem. 
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Now suppose that F : G — > G is split, and (T, A) is a pair in the G F -orbit 
corresponding to v_ G V". Apart from the full Wcyl group W(T), we will mainly 
be concerned with the F-fixed subgroup W(T) F and its subsets 

W(T) F = {we W{T) F | A o w = A}, 

W(T)f_> x _i ={we W(T) F | A o w = A" 1 }. 

We can consider these sets in the framework of §1.2 as follows. Define 

He) = {(Lj) g (a)\L x z 1 1 < j < tfo)}. 

By abuse of notation, write v_ also for the partition of n whose parts are 

( ra e(^)j)(«j)eAM- 

Then A(z^) indexes the parts of v_, in accordance with the notation of §1.2; also 
l(v) = |A(z/)| is the F 9 -rank of T. Now Q(j/) is the set of triples 

{(£,3,i) I (£,j) G A(i/),t e Z/m^i/^j-Z}, 

which is precisely the set of labels of the eigenlines of T used in §1.3. So W(T) 
in the above realization, namely as the group of permutations of £l(v), coincides 
with jS|„| in the realization of §1.2. As in that section, let w u G W(T) be the 
permutation (£,j,i) i— > {£,j,i + 1). Then by the description of F\t given in §1.3, 
W(T) F is exactly Z W ( T )\wv) = Z-- Note that 

e, = sign«) = J] €v( [] ("l)^- 
Je<<r>\L €e<<7>\£ 

For w £ W(T") F , we will use the notation u), i(w,£,j) G "Ljrn^iy^j of §1.2. So 
w(£,,3,i) = {w(€,j),i + i(w,£,j)), V(C,j) e A(^),i g Z/m^^jZ. 

With this notation, w G W(T) F lies in W(T) F if and only if w(£,j) = (£,/) 
(for some j') and m^ i(u;,£,j) hold for all (£,j) G A(z/). Here it is helpful to 
consider Ufe(<T>\L ^(^?)> which is the set of triples (£,j,s) with (£,j) G AQ/) and 
s G Z/(z^)jZ, and Ilfe<o-\u, ^kjh which is the group of permutations of such triples 
which preserve the first factor. For w G W(T) F , we define w G Ilfe^u, ^Vd by 

-ct • ^ r-re -\ , i ( w '£>J)\ 



rri£ 
F 



Clearly w i— » w is an isomorphism between W(T) X and Ilfe^YL Z v ^ . 

The analogous description of W(T) F ^ X _ 1 is as follows. For £ G (cr)\L, let £ v be 
the chosen representative in the (cr)-orbit of £ _1 . Define io(£) G Z/m^Z by 

e v =£7 io(£) (r i )) 



so that «o(£ v ) = — io(0- Assume v^v = ^ for all £ G (<j)\L; otherwise W / (T) F ^ A _ 1 
is empty. Clearly w G VF(T) F lies in W{T) F ^ x _ t iff w(£, j) = (£\ j') (for some 
j') and i(w,£,j) = io(Q mod m^ hold for all (£,j) G A(i/). For every permutation 
w of A(z/) such that w(£,j) — (£ v , j') (for some j'), and every (£, j) G A(^), lift 
«o(0 to an element io{w,£,j) of Z/mf(f{)jZ in an arbitrary way. Consider the set 
of permutations of Ofe<<r>U. ^(^f) which interchange O(zz^) and fi(^v) for all £. 
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Since v^y — v^, we can identify this set with Yise f v > ^W|- For w ^ ^(-Oa-^a- 1 ' 
we define w e ri{{^vj S^ by 

171(1 

Clearly w \- ► w defines a bijection between W(T)^_ >A _i and rj{£ «) Z v * . 

Now consider involutions in W(T) F . Since W{T)f m = Z^ v , we can apply the 
concepts of §1.2. In particular, any w & W(T)( m decomposes A(V) into 

1 



KdO = {(^i) e A (^)|w(£,j) = {Z,j),i(w4,j) = 2 m ?(^)j}' and 



V 2 
A^fe) = {(^i)eA(^)|^,i)/(e,j)}. 

If w e VF(T)£_ inv , then A£,(j/) is empty. As in §1.2, we write i l w {v) for |A^(V)|, and 
also use the notations £ % w (v)o, l l w (v)i, ^,(^2mod4 etc. Note that VF(T)±_ inv , i.e. 
the F-fixed points of W(T)±_ inv = (5 , i^i)±_i nv , is identified with Zj_ inv as defined 
in §1.2, and similar statements holds for W(T)f + -\_ inv and VF(T)f_ inv . 

We will aso write W(T)£ inv for W(T)f n W(T)[ nv , and V^(T)^ A _ 1 inv for 
W(T)^ x _ t n VK(T)£ V . Under the isomorphism W{T){ ^ FJ ? ^ defined above, 
W(T)^ inv corresponds to FL Z"* v . Moreover, as will be crucial later, if w E 
W(T)^ inv corresponds to (w^), then 

te = E £ ^)> 

«G<<t)\L 

and similarly for £ 2 and £ 3 . However, the situation is more complex when we 
introduce divisibility criteria, because of the division by m^: for instance 



too= E 4 £ K)o+ E *i.«te)- 

2fmj 2|m^ 



To get a similar description of M / (T) A ^ A _i inv , we need to put further constraints 
on the choice of io(w,£,j). It is easy to see that we can arrange to have 

io{w,w(£,j)) = -io(w,€,j) 

except in the case when £ v = £, £ ^ 1,-1, 2|(^)j, and w(£,j) — {£,j)- (For 
instance, when £ v = £, £ ^ 1, — 1, 2 \ {v()j and w(£,j) = (£, j), we are forced to 
set io(w,£,j) = ^m({v()j, since io(0 — h m €-) But if w is induced as above from 
w G W(T)^ X _ 1 . , this case cannot arise, since if 2|(f^)j we cannot have both 

i{w,€J) = -i(w,£,j) mod m ? (^)j, and 

*0,£, j) = «o(0 = 2 TO e mod m e- 

Sotii^tti with these conventions is a bijection between W(T)^_ t ^_ 1 inv and 

C >< CTv 1 X II { W « e Cv I 4, (^)O = 4 5 (^)O - 0} X J] Z£ . 
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In contrast to the previous situation, if we write w as (w^) £ Yiie f v | -^inv then if 
£ v ^ £, the elements of A* (z/f) contribute not to t\,{v) but to l%(v). Similarly, 
if £ v = £, £ 7^ 1, —1, then A^, Au(:) conributes to £%(v)- This subtlety is one factor 
complicating the formulas below. 

Everything we have said applies equally well to the non-split case, with a instead 
of a and rh^ instead of m^ throughout, except for the fact that the F q -rank of T is 
not l(v) but ^(V)o- Indeed, q — 1 (as a polynomial) divides 

\(k x y m ' iiv ' i)j I = q™*( u s)i - (-i)™«( l/ «)3 

once if rh^(v^)j is even and not at all if ifi^{v^)j is odd. In general, the main 
difference between the split and non-split cases below lies in the calculation of 
F g -ranks, which affects the signs in Lusztig's formula. 

2. Cases where G/K = GL n /Sp n 

In this section, we suppose that V has a nondegenerate symplectic form (•, •) (so 
in particular n is even), and that 9 : G — » G is the involution defined by 

{e(g)v,v') = {v,g- 1 v l ), Vg£G,v,i/eV. 

Since G e — Sp(V, (•, •)) is connected, K must be equal to it. So G/K is the 
symmetric space GL n /Sp n . 

Let T be any maximal torus of G. In §1.4, we associated to any / G Oy an 
involution wj £ W(T), depending only on the double coset TfK. Since (•, •) is a 
symplectic form, every line is orthogonal to itself, so Wf is fixed-point free (as a 
permutation of the eigenlines of T, not as an automorphism of T). Thus 

Tn.fK.r 1 = {t£T\w f (t)=t- 1 } 

is connected, and Z G (T D fKf' 1 ) = T. Moreover: 

Proposition 2.0.1. The map f t-> Wf induces a bijection T\Qt/K ^> W(T)s^[ nv . 

Proof. This is very well known, especially when translated into the language of flags 
via the connection mentioned in §1.1. Surjectivity can be proved by an explicit 
construction, and injectivity is easy by induction. □ 

2.1. The GL n (W q )/Sp n {W q ) Case. In this subsection, let F : G -> G be a split 
Frobenius map which commutes with 9. So F is induced by a Frobenius map Fy 
on V which respects (■,■). One has G F =* GL n (¥ g ), K F S ^(F,). The following 
result was obtained by a different (and simpler) method in [|l| §4]: 



Theorem 2.1.1. For any p £ V, 



n • 



( V P TnH GL "( F «Vn\ - / 1 ' l f all pe, are even 
\X.-^^s Pn {w q )^)l | o, otherwise. 

By the results in §1.3, it is equivalent to prove that for any y_ £ V° 
(2-1.1) ( J B,,Indf F (l)>= [] Yl & 

£6(<r)\i^l-|^| 
P£ even 



Note that the sign in (|l.3.1) disappears since n is even, and \p\ is even for even p. 



We aim to deduce this from Lusztig's general formula for the left-hand side 



(Theorem 1.1.1 above). This entails successively analysing the components of the 
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formula in our combinatorial terms, until we are reduced to a fact about class 
functions on the symmetric group (in this case one which is well known). This 
basic strategy will be repeated in every case; the main reason for including the 
present case, where the result is not new, is that it serves as the prototype for the 
following subsections. 

Let T be an _F-stable maximal torus, and A : T F — > Qj a character, for which 
(T, A) is in the G F -orbit corresponding to v <E P% . We will use the description of 



W(T) F as Z^- given in §1.4. Proposition |2.0.l| implies: 
Lemma 2.1.2. The map f *—> Wf induces a bijection 

T F \Q F /K F ^W(T)%_ inv 



Proof. Clearly the map in Proposition |2.0.1 is F-stable. So we need only note that 

o^™ T IS n „A »11 Tn fl*' t — l n y n ™^„^ + ^ r TF\ O-F / TS F _ ( rp\ Q_ / TS\F \~ 1 



since T, K, and all T n fKf^ 1 are connected, T F \Q%/K F = {T\0 T /K) F . D 

Lemma 2.1.3. For f € 9f , / € 9f A «■ W/ € W(T)f . 

Proof. Since T n fKf- 1 = {teT\ w f (t) = i^ 1 } is connected, e T ,/ = 1, so 

q t,a = {/ e t I ^l{teT F |« I/ (t)=t- 1 } = !}• 
Thus it suffices to show that 

{teT F \w f (t)=t-i} = {tw f (t)- 1 \tET F }. 

This follows easily from the fact that wj G W(T) F _- mv . D 

Corollary 2.1.4. TTie map / t— * Wf induces a bijection 

T F \Q F >X /K F - W(T) F >3 _ inv . 
As noted above, Zq((T n fKf~ l )°) = T, so Lusztig's formula becomes 

(%ind^(i)) = |r F \e|: A /^ F | = |^(T)f iff _ inv |. 

It is clear that under the isomorphism W(T) F ^> Y\.ee(a)\L ^^^ gi vcn m §1-4, 
W(T) F s _ inv corresponds to Fife (<t)\l ^ff-inv Now we a PPly the combinatorial 
fact (for which see @ VII. (2.4)]): 

(2-1.2) |^_ inv | - J2 XS- 

p even 

This gives Equation ( |2.1.l| ) and hence Theorem |2 . 1 . l| . 

2.2. The U n (¥ q 2)/Sp n (¥ q ) Case. Now we keep the assumptions from before §2.1, 
but take F to be instead a non-split Frobenius map which commutes with 9. So 
OF is a split Frobenius map as above, induced by a Frobenius map Fy on V which 
respects (•, •). One has G F = U n (¥ q 2) 1 K F = Sp n (¥ q ). In this case the result is: 

Theorem 2.2.1. For any p E V°. 



{ X-~> im Sp n (* t )W)-'[ o, otht 



1 C r „(F q2 ) / , 1 ^ _ J 1, if all p^ are even 
herwise. 
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It is equivalent to prove that for any v_ £ P%, 
(2-2.1) (B,,Ind^(l))= [] Yl A- 

P£ even 

Note that the sign in ( 1.3.2 ) disappears because n(p') = '-^ mod 2 for any even p, 
so that 

E ™>tn(p'z)= E ™«-y- = ^mod2. 



Equation (2.2.1) is deduced exactly as in the previous section, using the non-split 



versions of the notation in §1.4. 

3. Cases where G/K = GL n /(GL n + x GL n -) 

In this section, we suppose that 9 is an inner involution, namely conjugation 
with respect to s £ G such that s 2 = 1. Let V + be the (+l)-eigenspace and V~ the 
(— l)-cigenspace of s on V, so that V — V + ®V~ . Let n + = dimy + , n~ = d\vnV~ , 
so that n = n + + n~ . Since G e = GL(V + ) x GL(V~) is connected, K must be 
equal to it. So G/K is the symmetric space GL n /(GL n + x GL n -). 

Let T be any maximal torus of G. For / £ ®t, we have Wf £ W(T)- mv , not 
necessarily fixed-point free. Indeed fixed points of Wf correspond to eigenlines of T 
which are stable under s, and therefore lie in either V + or V" . Let 

€f : {fixed points oiwf} — ► {+, — } 

be the resulting map. In the notation of §1.2, (wf,Cf) £ W(T)r n + n -\ inv . 

Proposition 3.0.1. The map f \— > (w/,ef) induces a bijection 

T\Q T /K^W(T) {n+fn - } _ im . 



Proof. As with Proposition 2.0.1, this is well known when translated in terms of 



if-orbits on the flag variety, and easy to prove (see for instance |lq] ). □ 

3.1. The GL n (¥ q ) / (GL n + (¥ q ) x GL n - (¥ q )) Case. In this subsection, assume that 
F : G — > G is a split Frobenius map such that F(s) — s. So F is induced by a 
Frobenius map Fy on V which stabilizes V + and V~. One has G F = GL n (¥ q ), 
K F = GL n +{¥ q ) x GL n -(¥ q ). Recall the definition of T^+^-^p) from §1.2. 

Theorem 3.1.1. For any p £ P%, 

f IT C Ml */Pf = /°?- 1 >^ 

(x^lnd^ZiuGL cfA 1 )) = \ |i(n+ '"- )lPljl ' andp'_ x lS even 



l GL n+ (Vg)xGL n -(F q y 



0, otherwise. 



(Note that |^(n+,n-)(/ : 'i)l could be zero.) By ( 1.3.1 ), it is equivalent to say that for 
any !/£?>%, 

(S,,Ind^(l)) = ( E \T(n + ,n-M)\x P »l)( E X»-\) 

Pl^Wll p_ir-|l/_l| 

p_i even 

(3 - L1) >< n ((-D kd e A) n v**,**- 
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Here the sign (-!)"+£ ?e<<,>\£ l"«l = (_i)i: 5e <„>\L(™?+ 1 )l^l m ( [1.3.1] ) i s simplified by 



noting that mi = 77i_i = 1, mj is even for £ ^ 1,-1 such that £ v = £, and z^ and 
V| 2 must be equal whenever Ci 7 = £2, £1 7^ £2 in order for the right-hand side to be 
nonzero. 

Let (P, A) be a pair in the G F -orbit corresponding to y_ € P^. 

Lemma 3.1.2. T/ie map / 1— ► (wf,ef) induces a bijection 



T F \Q F /K F ^W{T) F n+ ^ 



)— inv* 



Proof. Again, this follows from Proposition 3.0.1, since P, K, and TDfKf 1 = T 



»/ 



are all connected. □ 

Note that for w € W(T)f_ >A -i,hiv' &i) e A ^ ** £ = ±L Dcfinc 
F4 = {^ g W(T)^ A _ Iiinv I (U) 6 Ai(i0 => £ = 1}. 

Lemma 3.1.3. For f 6 9f , / € 6£ A <S> w/ e 5^ v . 
Proof. Since P n fKf- 1 = T w f is connected, e T j = 1, so 

©?,A = {/ee?|A| (TPr/ =i}. 

Thus it suffices to show that [T F ) Wf is generated by 

{««;/(*) 1 1 € P F } and JJ (fc*)^"^ . 

(^)eAJ, / (jO 

This follows easily from our description of T F . □ 



Corollary 3.1.4. The map / 1— » (wf,Cf) induces a biject 

T F \Q F JK F ^*f + , . . 

\ i 5 A/ (n+,-n ) — inv' 

where Yr~ + _x . = {(itf, e) G 2r + _x • I u; G K~ }. 

(n + ,n )-inv LV ' ' (n+,n )-inv I inv J 

Now the Fg-rank of T is l{v), and that of 

z G ((TnfKr 1 )°)= II Gi ( L («^)) 



ion 



x J| Gi(p (Cj - l) eP (Cj - l+ i m!:( ^ )j) ) 



is ^(i/) + £^L (£). So Lusztig's formula becomes 



(^,indi; ( i))= y: (-ir^= E ho* 00 - 

f£Tr\e'/K*> 
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Under the bijection w <— > w defined in §1.4, Y-~ ■ corresponds to 

ie(a)\L { ?17 t ?2 } G ( CT )\L 



S^i-i 



Hence 



(^,Indf,(l)) = £ (-l)^(-) (e,_J4-4 nv |) 



v («'i.«)6^ + ,„- ) _ 



J] (-i) l ^ l IKe^l^ 5 (^)o=e 5 K)o = 0}| 



€ V =C 

X 11 S »ZiV<L 2 Zv Sl- 

{£iJ±S2}£(<r)\L 

&=& 

In the second factor we have used the fact that if W-\ £ ZgZ.\ nv , then 
4_>-i) = *L>-i)o = *(«>-i)o mod 2. 



In the third factor, we have noted that both A* (z^) and A^ (i/^) contribute to 
(-w(e)' an d if there exists w^ £ Z v * such that £* (^)o = ^L ( t/ ?)o — Oj tnen 



tl t (yt)+%, t M s ^ - 1^1 mod 2 - 

In the fourth factor there is no contribution to l%,{v), as noted in §1.4. Now we 



apply (2.1.2) and the following combinatorial facts (for which see §5): 



(3-1.2) J2 (-!) CM = E iv^^oix 



'W^P 



(t "' E)eZ ( P +,p-)-inv PHM 



(3.1.3) \{w e zr nv |^Ho = 4 2 »o = o}| = E x£- 

pl>l 

These give Equation ( |3.1.l| ) and hence Theorem |3.1.l| . 

3.2. The GL n (¥ q )/GL n / 2 (¥ q 2) Case. Now keep the assumptions from before §3.1, 
but take F to be a split Frobenius map such that F(s) = —s. (Then F still 
commutes with 8.) So F is induced by a Frobenius map Fy on V which interchanges 
V + and V~ , whence n is even and n+ = n~ = j. One has G F = GL n (F q ), 
K F S GL n/2 (W q3 ). The result is: 

Theorem 3.2.1. For any p E V°, 

I p T ,gl„(f,) /-.xv _ J 1, «/P5 = p ? -i,V£ ; pi is even, and p'_ 1 is even 
\X-> ma GL n/2 (t q2 )\ L )) ~ | 0] otherwise. 
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By ( J1.3.1D , it is equivalent to say that for any v e V, t 

<S,,Indf,(l)) = ( Y. *£)( E XlZ\) 
(3.2.1) 



PiH|ial| 

pi even 


p- 
p 


_ 1 even 


1 














>< n 


((- 


.1)KI 


E 


x p 4) 


n 




5 "«, 


"«2 


2i/ 4l 


t£(<r)\L 






P5 h ks 




{?i#«2}e( 


7>\i 









(For the signs here, see the comments after Equation ( |3.1.l| ).) 

Le t (T, A) be a pair in the G F -orbit corresponding to y_ £ P°. Obviously Lemma 
3.1.2] becomes: 



Lemma 3.2.2. The map f <— ► (wf,€f) induces a bijection 

T F \Q F /K F - iy(T)f„ inv . 



Now Lemma 3.1.3 holds again here, with the same Yj- v and the same proof. So 



arguing as in §3.1, we get 

<^,Indf,(l)>= EM)^' 
(w,e)eyf_ inv 

where the definition of YjJ_ inv is the obvious one. The rest of the proof is also the 



same as in §3.1, except that Equation (3.1.2) is replaced by 



(3-2.2) E i- 1 ^ = E #■ 



(w,e)e.Z"_, pHH 

p even 



This too will be proved in §5. 



3.3. The E/„(F g 2 )/([/„+ (F q 2) x U n -(¥ q z)) Case. Still under the general assump- 
tions of this section, let F : G — > G be a non-split Frobenius map for which F(s) = s. 
Replacing s by a G F -conjugate if necessary, we may assume that there is some non- 
degenerate symmetric form (■, •) on V, for which V + and V~ are orthogonal, and 
such that the associated outer involution 8' : G —> G commutes with F. Then O'F 
is the split Frobenius map induced by some Fy : V — ► V which respects (■, •) and 
fixes V+ and V~. One has G F £* C/„(F g2 ), K F = U n+ (¥ q2 ) x [/"„_ (F q2 ). 

Theorem 3.3.1. For any p G P%, 

n <w i f IT (VVAl ^ =Pf- I >V^ 

(x^,Ind^ ( ^ 2)x[/n _ (Fg2) (l)) = <^ l J (» + .»-)^^ I' ondpLiMewen 

I 0, otherwise. 



AT' -INVARIANTS IN IRREDUCIBLE REPRESENTATIONS OF G" , WHEN G = GL n 19 



(Note that \T( n + ,n~){p'i)^'\ could be zero.) By ( 1.3.2 ). it is equivalent to say that 
for any v G P%, 

(3.3.1) 
(B,,Ind^(l)) = ( J2 (-^M^^-M^M- 1 ) 1 -^ 11 E X P uZl) 



ll-kll 






p_ 1 oven 


>< n ((- 


-1) 


**' E x3) 


n ( e x«) 


4|m 5 




p« h kd 


{S<«3r)\L p c l-|^| 

«^1-1 
4fmj 


n 




°"h"ii Z "ti 


n (-i) 1 ^ 1 ^^^- 


2|m ei 


)\L 


{6#6}e(*)\L 

2fm Sl 



Here the sign (_l)rtl+i:« e < 5 )\i>€™(4)+lpel of |^^ ig simplified as follows. If 
/?£ = p^-i for all £ and p'_i is even, then 

n = y rh^\p^\ = \pi\ mod 2, 
and the sign can be replaced by 



(-l)L^ Li J+«(pi)(_i)^2^+™(p'-i) TT (_i) 



«e<*)\L 



TT (_l)m« 1 (IP€iH- 2 "(4i))+ 2 l' : 'Sil. 



{Ci#6}G(a>\L 

Then we observe that if p'_]^ is even, n(//_i) is even; and if all even parts of p[ occur 
with even multiplicity (which is necessary for a(n + , n - , p[) / 0), then the Young 
diagram of p[ (excluding the top left corner if \pi\ is odd) can be tiled by 2 x 1 
dominoes, from which we see that [^j J + n {p'i) + n {pi) is even. 

Let (T, A) be a pair in the G^-orbit corresponding to y_ g V^ . Proposition J3.0.1 
has the following corollary in this case: 

Lemma 3.3.2. The map f i— > (wf,ef) induces a bijection 

t f \b f /k f ^COf n+ , n -)_ im , 

Define Y^ v C W r (T)^ A _ 1 inv in the same way as in §3.1, but in the a version. 
Lemma 3.3.3. For f € 9f, / € 6f A <£► u;/ € 5^ v . 



Proof. The proof is exactly analogous to that of Lemma |3.1.3| . □ 

Corollary 3.3.4. The map f ^ (wf,€f) induces a bijection 

(n+ : n ) — inv 



T F \@ F /K F ^Y^ 
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The important point of difference from §3. 1 is the F g -ranks involved. The ¥ q - 
rank of T is now £(v)q, and that of 



Z G ((Tn.fKf- 1 )°)= J] GL(L ( ^ 



3,i)> 






x n GL ( L (U,i) ® L (U,i+h™e("eh)) 

{(S,J,i),(«J'.*+2™«K)j)} 

is t(v) Q + 1%, (v)amo&i + \P-w s {k)i- (The GL 2 factors corresponding to (£,j) G 
A-wAll) are split iff 4\m^(v^)j, and those corresponding to {(£, j) ^ w/(£, j)} are 
split iff 2|m^(fc'{)j.) So Lusztig's formula becomes 

(^,Indf F (l)) = 2 ( _ 1) ^ / (i£)o»o d4+ i^ / (^) 1 



(«'.«)e^ +in _ ) _ lnv 
Using the same reasoning as in §3.1, we can transform this expression to get: 



(B,,Ind^(l)}= £ (-l^W-^+^il" 1 )' 

( t0 i,ei)ez ( " n 1 + _ 7i _ ) _. nv 

x (-i^^KlU 

x J] (-i)^|{^eCtl4 5 (^) = e 5 K) = 0}| 

4|mj 

x [] IKeCH4,(^) = ^(^) = o}| 

«6(<J}\L 

c v =e 

4-fm^ 

x n ^^ n (- i )' (v<i)i ^ 1 ^^ 1 - 

2|m 4l 2fm £l 



In the second factor, we have noted that if W-i G Z s _\ nv , then 

W-i\ 



Ci( i/ -l)0mod4 + 2^-i(^-l)l = ^-l)omod4 + --£(zA-l)l 



^(^-1)0 mod 2. 
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In the third factor, since 4|m^, both A^ (i^) and A^, (y^) contribute to ^(V)omod4, 
and nothing contributes to £^,(u)i. In the fourth factor, since 2|mj but 4{ rh{, and 
^wf( v i)o — ^w ( J/ ?)o = 0, there is no contribution to the sign. So in addition to 
(2.1.2) and ( 3.1.3| ), we need the following fact: 

(3.3.2) ]T (-l)<»o»=- ^''"(^^^(-ir^lV-P-J^lxS- 

(u, ' e)ez (;+, P -)-inv ^M 

This will be proved in §5. 

3.4. The U n (F q 2) /U n /2(J$ q i) Case. The final case to consider in this section is 
when F : G — > G is a non-split Frobenius map for which F(s) = —s. Replacing s 
by a G^-conjugate if necessary, we may assume that there is a form (•, •) on V and 
an involution 6' : G — > G with the same properties as in §3.3. Then 8'F is the split 
Frobenius map induced by some Fy : V — > V^ which respects (■, ■) and interchanges 
V^ + and V~. In particular, n is even, and n + = n~ = §. One has G f = C/ n (F g 2), 
^ F = £4/2 P^V). The result is: 

Theorem 3.4.1. For any p G P%, 

{if Pi =P£-i.V^, 
n(m*(pi) + l), 2|^ 9 H- 1 (pi),Vi 1 
« and /9l x is even 

0, otherwise. 



By (1.3.2), it is equivalent to say that for any v_ e V", 
(3.4.1) 
(B^Indg^l)) = ( E (- 1 ) n(Pl) (II(^(Pi) + l))xg) 



2|m 2 «+i(p'i) 

p-lH^-ll 

p'_ 1 even 



x((-i)^ E xfci) 



X 



n ((-!) kd e x«) n ( e xj?) 

4|m{ 4-fm^ 

n v^^i n (- i ) k€ii ^«i^ 2 ^i- 

tf=6l e x v =6 

2|m Sl 2|m £l 



For the signs here, see the comments after ( 3.3.1 ). 
Let (T, A) be as in §3.3. We have: 

Lemma 3.4.2. The map f \— > (wf,ef) induces a bisection 

T F \Q F /K F Z W(T) F _ inv . 
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Now Lemma 3.3.3 holds again here, with the same Yj~ v and the same proof. So 
arguing as in §3.3, we get 

(A,,Ind^(l)> = ]T (_l)^(K)om.d*+i4fe)i. 



The rest of the proof is also the same as in §3.3, except that Equation (3.3.2) is 
replaced by: 

V (_l)'2»0mod4+^(")l 

(3A2) = £ (-i) nW (II(^^) + i))x^ 



2|m 2 i+i(p') 



This will be proved in §5. 



4. Cases where G/if = GL n /O n or GL n /SO n 

In this section, we suppose that V has a nondegenerate symmetric form (•,•}, 
and that 9 : G — > G is the involution defined by 

(%>,«'} = <«,<rV}> v$eG,vyeV. 

Since G 9 = 0(V, (•, •}) has two components, K can be either O n or SO™. It suffices 
to solve the problem for O n : 

Lemma 4.0.1. Let F : G — » G 6e a Frobenius morphism which commutes with 9. 
Let p £ V^ if F is split, p £ V^ if F is non-split. If F is split, let ( £ L" be such 
that (-1,C) CT = -!■ If F is non-split, let ( <E L a be such that (-1,0* = -!• ^Zien 

( X £,Ind^ e)0)F (l)) = (x^Indf G V(l)) + ( X C ^,Indg»)-(l)). 
Proof. Clearly 

Ind[J]" 0)F (l) = Resf^ (1 + (det(-),r 1 )), 
so 

Ind^)o)^(l) = (l + (det(-),r 1 ))-Indf G F fl)F (l), 
which proves the result. □ 

For the remainder of this section, we will write K for G 9 = O n and K° for (G e )° = 
50„. 

Let T be a maximal torus of G. For / £ 0t, we have u>/ £ IF(T)i nv . This time 

Tn/if/- 1 = {teT| W/ (t) = i- 1 } 

is not necessarily connected. 

Proposition 4.0.2. The map f \— ► Wf is a bijection T\Qt/K — > VF(T)i nv . More- 
over, if w £ VF(T)j nv; t/ie corresponding T-K double coset breaks into two T-K° 
double cosets if w £ IF(T)ff_i nv; and is a single T-K° double coset otherwise. 

Proof. As with Propositions 2.0.1 and [3.0.1 , this is better known as a statement 
about -ftT-orbits on the flag variety (see 15, §6]). It is easy to prove. □ 
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2:i 



4.1. The GL n (¥ q ) / O n {¥ q ) Case (n odd). In this subsection, suppose that n is 
odd and let F : G — > G be a split Frobenius map which commutes with 9. So i* 1 is 
induced by a Frobenius map Fy on V which respects (-,•), such that (•, •) has Witt 
index LfJ on V Fv . One has G F =* GL n {¥ q ), K F = O n (¥ q ). Recall the definition 
of d^ from §1.2. The result is: 

Theorem 4.1.1. For any p G P%, 



( y p i n H GL »( F «) 



(1)) = 



5 n (iiK(' o ?)+ i ))' v d z=- 1 



H 



is even 



Z£{a)\L 

d i= i 



0, 



otherwise. 



By ( 1.3.1 ), it is equivalent to say that for any v_ s V%, 

(B,,Ind^(l)) = -i [] ((- 1 ) 1 " 51 E (II( m *^) + 1 ))^) 



?e( CT )\i 



PsH^I 



(4.1.1) 



n ( e x8). 

p'i 



Here the sign in (1.3.1) has been distributed in an obvious way. 
Let (T, A) be a pair in the G F -orbit corresponding to v_ € T^ 

Lemma 4.1.2. TTie map / i— ► wj induces a surjection 

T F \e F /K F ^W(T) F nv . 



Proof. By Proposition 4.0.2 and Lemma 4.1.2 , we may identify W(T) F 1V with (T\ 
Qt/K) f . Under this identification, an involution is in the image of T F \Q F /K F 
precisely when the corresponding F-stable K -orbit on T\Qt contains an F- fixed 
point (by connectedness of T). Since n is odd, this i^-orbit is a single i"f°-orbit, so 
this is automatic. □ 



2 £ V- ) 1 T F -K F double cosets in (TfK) 1 



Lemma 4.1.3. For f 6 Q F , there are 

Proof. Since the image of the Lang map on K is K°, the number of T F -K F double 
cosets in (TfK) F is the same as the number of orbits oiTnfKf^ 1 on TnfK°f~ 1 
for the action t.t' — tt'F(t) . Using Lang's theorem, one sees that only the 
"disconnected part" of T n fKf^ 1 contributes. So this is the same as the number 
of orbits of 



(U)eK f dd 



n e \u,i) 



TT {±ir d^h on <j (e ' (o . , )} e TT (± i)»^(^), 

for the action (e({ J ,i))-(e / ( ?>i , i) ) = ( e (£j,*) e K,i.i-i) e (fj,»))' In otllcr words ; [t is tne 
index in the latter group of the subgroup where rit e rf a) — L f° r au (Cii) S 
A 1 ,(l£). Now since n is odd, A 1 (z/) is non-empty. The result follows. □ 



Now let *£ v = {«, e W(T)f . | (£,j) G AiOd) =► (-1,0' 



= !}• 
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Lemma 4.1.4. For f G 6£ ; / G 0£ A 4^ it?/ G X^ v . 

Proof. We first note that er,/ = 1, as may be seen directly (using formulas for 
F g -rank such as those below) or deduced by the method of [[10| Lemma 11.3]. So 

@t,a = {/ e ©t I M{ter F \w f (t)=t- 1 } = !}• 
Thus it suffices to observe that {t G T F \ Wf(t) = t -1 } is generated by 

{tw f (t) \t GT F } and [] (±1). 



(^eAJ^fe) 



□ 



Corollary 4.1.5. TTie map / i— > w/ induces a surjection T \Q F X /K -» X[ 



Now the F q -rank of T is £(v), and that of 

/ 



Z G ((TnfKf- 1 )°) = GL 



\ 



L («j,0 

x n Gi ( i «^))x n Gi ( L (?^) 



foOeA^Q/) 



KjOeA^Q/) 



IK 



%)+ E KK) J -l) = %)+« + 4 / (^)mod2. 






So Lusztig's formula gives 

(B,,indg;(i)) = (-ir e (-i) 4 '^ 



f£T"\eFjKt 



toGXr^ v 

-J E (- 2 )^- 



Under the isomorphism W(T)£ -^+ llfe( CT )\L ^"* °f §1-4, X^ v corresponds to 

{K)g J] ^vK = -1^4,(^)1 = 0}. 

Hence 



/ 



<B,,Indf F (l)) = -i n 



«e( CT )\i 



\ 



E (-2 



/i>«) 
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2~, 



So to prove ( |4.1.l| ), the combinatorial facts we need are: 

(4.1.2) ]T (-2)'» = (-1)1-1 J2 ([[(Mp) + !))*£, and 



wEZ" 



(4.1.3) 



p\-\v\ 


I 


E(-2) 4M = 

wez? m 

i»i=o 


- E xt 

P \-\ v \ 

p' even 



These will be proved in §5. 

4.2. The GL n {¥ q ) / 0^{¥ q ) Case (n even). In this subsection, suppose that n 
is even and F : G — ► G is a split Frobenius map which commutes with 0. So 
F is induced by a Frobenius map Fy on V^ which respects (•, •), and once again 
G F ^ GL n (¥ q ). The Witt index of (-, •) on V Fv is either § or § -1, and accordingly 
either K F = 0+(¥ q ) or K F = 0~{¥ q ). Let e be 1 in the first case and -1 in the 
second case. 

Theorem 4.2.1. For any p G P%, 



(x^M4ifj\i)) 



\ n ([[(™i(pc)+ i ))> ifd^-i^p^ 



t£(a)\L i 



d f =\ 



0, 



othe 



if all p'c are even 



0, otherwise. 



By (1.3.1), it is equivalent to say that for any v_ € P°, 



«e( CT )\i 



de=l 



P&\ v i\ l 



(4.2.1) 



n ( e A) 



ie(a)\L Pi h\v s \ 



d f = -l 



Pi 



n ( e x p s 



is even 



(B„,Tndf P (l)) = l [] ((- 1 ) 1 "* 1 E (U(™i(Pt)+l))X 



ie(a)\L Pi \-\ Vi \ 
p* even 

Let (T, A) be a pair in the G F -orbit corresponding to y_ £ P°. 

Lemma 4.2.2. The map f t— > Wf induces a map T F \Q F /K F — > W(T) F n 
e,, = e, this map is surjective. If e^ = — e, the image is W(T) F nv \ W(T)g_ inv . 



If 



Proof. As in the proof of Lemma 4.1.2, an involution is in the image of T F \Q F /K F 
precisely when the corresponding instable if-orbit on T\0r contains an F- fixed 
point. If the involution has a fixed point, this if -or bit is a single K °-orbit, so this 
is automatic. Suppose w G W(T)^_ inv . It is in the image of T F \Q F /K F precisely 
when there exists a decomposition of V into lines {L^ 1 1 < i < n} such that 

1. (L^ =@v#wwL<„ and 

2. F V (L$ = L' {i) 
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(here we have identified W(T) with S n in some way). We must prove that this 
happens if and only if sign(uv) = e. Since e is multiplicative with respect to in- 
stable orthogonal direct sums, we may assume that (w) x (uv) acts transitively on 
{1, • • • , n}. So if sign(uv) = 1, then (uv) has two orbits on {1, • • • , n} which w 
interchanges; in this case the existence of {L^} as above is clearly equivalent to the 
existence of a decomposition V = V\ @ V2 into Fy-stable Lagrangian subspaces, 
which indeed happens if and only if e = 1. On the other hand, if sign(uv) = — 1, 

then w v is an n-cycle and w = uv ; in this case the existence of {14} as above is 
easily seen to be equivalent to the existence of a Lagrangian subspace V\ of V such 
that dimT4 H Fv(Vi) — § — 1, which indeed happens if and only if e = — 1. □ 

Lemma 4.2.3. For f G Q F , the number of T F -K F double cosets in (TfK) F is 

f ^ 1, i ifw f £W{T) F _. mv 

12"'" , otherwise. 



Proof. The method of proof of Lemma 4.1.3 applies again here. □ 

Now define X£ v as in §4.1. Let X%_ inv = Xf m n W(T) F _ iav . 
Lemma 4.2.4. For f G 6£, / G 6£ A &■ w f G X^ v . 



Proof. The proof is exactly the same as that of Lemma 4.1.4. □ 



Corollary 4.2.5. 1. The map f t—> Wf induces a map T F \Q F X /K F — ► X^ 
7/ e„ = e, i/iis map is surjective; if t v = —e, its image is X ; — v \ X^_ inv . 
2. 7/w is in the image of T F \Q F X /K F — ► Xj^ v , f/iere are 

I z / , otherwise 

T F -K F double cosets in the preimage of w. 



Proof. This follows by combining Proposition 4.0.2] , Lemma 4.2.2, Lemma 4.2.3 and 



Lemma 4.2.4. □ 



Now as in §4.1, the F g -rank of T is %), and that of Z G ((T n fKf~ 
congruent to t{v) + n + i^Av) mod 2. So Lusztig's formula gives 



A^) 



(i^Ind£ P (l)> = X! (- 1 ) 

feTF\Q* iX /Kr 

1, ifwGX 



iuGX- 



^ l j 1 2'i-W- 1 , otherwise 



ff— inv 



-5(1-^1x1^1 

1 ^ ( _ 2 )^) + ^^ 

toexL 
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We transform this expression as in the previous subsection to obtain: 



£g<<r)\L 



( J B i£ ,Ind|;(l)) = i [] 

4n 



E (-2) 



£«("«) 






p 1 7"« I 



CS(<t)\L 



In those factors of the second term for which m^ is even, we have used the fact 
that if there exists a fixed-point free involution in Z v s, then £(v^)i is even, so 



(-!)*("«) = e V( . So along with (|T|) and ( |4.1.3[ ), we need ( |2 . 1 . 2| ) multiplied on 



both sides by e„. 



4.3. The U n (¥ q 2)/O n (¥ q ) Case (n odd). In this subsection, suppose that n is 
odd and take F : G — > G to be a non-split Frobenius map commuting with 0. Thus 
^i* 1 is a split Frobenius map induced by Fy as in §4.1. One has G F = U n (¥ q 2), 
K F ^ O n (F,). The result is: 



Theorem 4.3.1. For any p e ^ 



(X^Ind^f ) ) (l)) = <j 



' \ n (ii( m ^ + i(^)+i)) n di( m ^(^)+i)) 

2\m$ 2fm 5 

x n (ii("»i(p«)+ 1 )). 

2|m £ 

i/ J 5 = 1, 2{m ? =>• 2|m 2i (p ? ), Vi, 
J ? = -1, 2{to 4 =*■ 2|m 2i+ i(p 4 ), Vi, 
and d^ = — 1, 2|m^ =>■ pi is even 
0, otherwise. 



2N 
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By ( J1.3.2D , it is equivalent to say that for any v€P° 
(4.3.1) 



( \ 

<S,,Indf F (l)) = -(-l)LfJ J] Y, (-l) n( ^ ) (II(^ + i(^) + l))xS 



d 4 =l \2|m 2l (p 5 ) 
2|m 5 



n e (- i ) n(4) (ii( m 2i(pe)+i))x^ 

d 5 =— 1 \2|m 2 i+i(p{) 
2{m ? 



n (- i ) ki ( e (ijtote)-*- 1 ))* 

Pf, h 1 1/ 5 » 



2|m^ 



>< n 



E xi\ 



d^ = — 1 Ypj even 



2 I 7 7 V t: 



/ 



Here the sign 

(-l)rfl+E 5 e{s>\i''"s n (p«)+lp«l _ (_nLfJ+IZ ?e (s>\t™«("(P{)+|p£l)+lpd 



of (1.3.2) has been distributed in an obvious way. 

The proof of these statements is mostly very similar to that of (|4.1.l| ). Let (T, A) 
be a pair in the G F -orbit corresponding to y_ € V%. 

Lemma 4.3.2. The map f i— ► Wf induces a surjection 

t f \q f /k f -» W(T)£ V . 
Proof. This is deduced in the same way as Lemma |4.1.2[ , D 



Lemma 4.3.3. For f E Q F , there are 2^"'/ ( - ) 1 T F -K F double cosets in (TfK) F . 



Proof. The proof is the same as that of Lemma 4.1.3 



□ 



Now define X-^ v as in §4.1, but with a instead of a and rh^ instead of m^. 
Lemma 4.3.4. For f e 6f, / e Gf A &■ w f e X^ v . 
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Proof. We first prove that erj — 1- The F 9 -rank of 



Z G ((TnfKf- 1 )°) = GL 



n GL (ku,)) >< n GL (hu,i)) 



(U)eK f (z) 



(€j)6AJ,,(e) 



is 



v\ E ^K^J+C.^+^^o. 



KjOea^h 



If t E (T n /i^/ X ) F has eigenvalue a(£j) on L^jq^, then 

zg(t)nz G ((rn/A7- 1 ) ) 



GL 



e * 



(JjV 



. (f^eA 1 ^) 

V q («,j) =i 



xGL 



J 



e £ 



(£,i,- 



. (^•)eA 1 / Ci i ) 

V Q (5,j)=- 1 



n Gi ( £ c^-,i)) x n Gi ( L (?,^ 



i)) 



(U)eA 2 Wf (u) 






has F 9 -rank which differs from that of Z G ((T n fKf- 1 ) ) by 



L2 E "»f( v f)jJ + L2 E '"{(^JjJ - L5 E ™eK)jJ- 

(U)eA 1 / 00 KJ)eA 1 / ao (SjOeA^Q/) 

Q «,j) =1 Q (5,j)=^ 1 

Since n is odd, J^i? j)eA 1 M ^ l c(^)j i s °dd, so this difference is zero. Thus erj 



1. The rest of the proof follows that of Lemma 4.1.4 



□ 



Corollary 4.3.5. The map f 1— > Wf induces a surjection T F \0?p \/K F ~* Xfa v - 

Now the F g -rank of T is t(v) , and that of Z G ((T n fKf- 1 ) ) is given above, 
whence 

F g -rank(T) + F,-rank(Z G ((T n fKf- 1 ) )) 

(S>i)6Ai, / (i/) 

77- L 

S kj + tlfbdo + C / (^)2mod4 + ^fe)l mod 2. 



30 ANTHONY HENDERSON 

So Lusztig's formula gives (compare §4.1): 



(B^lndZUl)) = -(-l) LfJ J2 (-l)^ 0+ ^ )2m ° d4+ ^» fe)l 2^. 



As in §4.1, we transform this expression to get 



(S,,Indg;(l)) = i(-l)LfJ 



/ 



n 

2fm 5 



E m 



X 4 ("«)0+^ 4 ("i)jn M d4+^ j ("«)l 2 e ls ("0 



/ 



n 



2|mj 



E (- 2 ) 



<L("«) 



i«=-i=>4 € ("«)i= 



/ 



In the second factor, we hagve noted that if rh^ is even, no element of A^, (i/^) 
or A^ (z/f) can contribute to ^(^)2mod4 + 2^tU— )i> and ever y element of A* (;/{) 



contributes to ^,(V)o- So in addition to (4.1.2) and (4.1.3), the combinatorial facts 
we need are: 



y^ (_ 1 ^LHo4<.M2 „,od4+t^;l>)i 2^i>) 



wezy 



(4.3.2) 



and 



(4.3.3) 



E (-ir (p,) (n("^ + i(p) + i))x£, 



2|m 2l (p) 



Y^ (-l) f »(' y )+ £ -(' / )2mod4 + |^(^)l2^(^) 






E (- 1 ) n(p,) (II(^(p) + i))xe- 



ph|f 
2|m 2 i+l(p) 



These will be proved in §5. 



4.4. The U n (¥ q 2)/0^(¥ q ) Case (n even). Finally, we suppose that n is even and 
F : G — > G is non-split. So 0F is induced by Fy as in §4.2, and we have the same 
dichotomy as to the Witt index of (■, •) on V Fv . Define e E {±1} as in §4.2, so that 
G F *U n (F q ,),K F *0* n (F q ). 
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Theorem 4.4.1. For any p G 'P,° 



' \ n (n^M+i)) n (n( m 2m(p«)+i)) 



4=1 

2fm e 



2fm £ 



^ ind S ( (F: 2 ) ) (i)> = <i 



IT (II^^ + l)): 



2|m e 

ifd i = l, 2 fm ? =*> 2|m 2 i+i(p^),Vi 7 
<% = -I, 2 f m ? =>• 2|m 2i (p^),V«, 
and dg = — 1, 2 | rfi£ =>• pi is ewn 
0, otherwise 



+ 



^e, if all p'c are even 
0, otherwise. 



By (1.3.2), it is equivalent to say that for any v_ € V^, 
(4.4.1) 

/ 
(B,,Ind^(l)) = -(-l)t J] Yu (-l) nCp « ) (II( m «(«) + 1 )) 






d 5 = l \2|m 2 i + i(p 5 ) 
2+m £ 



n e (-ir ( ^ ) (ii( m ^ + i(^)+i))x^ 

£G(ct}\L I p c h|j/ 4 | • 

d 5 = -l \2|m 2i (p s ) / 



CG<S)\L 
d = = l 
21m,; 



vP«H"sl 



n (- 1 ) 1 "* 1 1 e (ii( m *(«)+ 1 )) 



v p « 



n 



E x p 4 



d 5 = — 1 ypj even 

2|mj 



/ 



J« n <-d ! 

2fm 5 



E xsi 

\ p't even 



\ / \ 

n 



/ 



E x p 4 



Pin^il 



i" l 5 \Pj even J 



Here the sign 
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of (1.3.2) has been distributed in an obvious way (in the second term it has been 

rewritten as life (o->\l( — 1)™ 5_5_ since \p%\ and n(pi) are even if p'? is even). 

The proof of this is similar to that of (4.2.1). Let (T, A) be as in §4.3. Lemma 
4.2.2J must be modified as follows: 

Lemma 4.4.2. The map f i— ► Wf induces a map 

T F \G F /K F - W(T) F nv . 
If e„ = (— l) T e, £/ws map is surjective. If e„ = — (— l) T e, i/ie image is W(T)^ V \ 

Proof. The proof is mostly the same as that of Lemma 4.2.2. Note that if / G O^, 
Fv{r l L {u , ) )=9{f)- 1 f) Lf e ^ V) 

( V 

r^«'j',i') 






/ 



f-lr 

■< «>, 1 w !L (£,j,i)' 



So in (2) of the proof of Lemma 4.2.2] , uv should be replaced by 
sign(w) = (—1)= if w G W(T)g_ inv , we get the result. 

Lemma 4.4.3. For f G Q F , t/ie number of T F -K F double cosets in (TfK) 1 



□ 



v..s 



2 4,G0-i 



z/ W/ G^(T)£_ ir 

otherwise. 



Proof. The proof of Lemma 4.1.3 applies again here. 
Now we diverge somewhat from §4.2, and define 



Xt 



{w e w^(T)f, nv | (c,j) g Ai,oo =► (-1,0'" 



(-l)™^^}. 



n 



Lemma 4.4.4. For / G 6£, / G 6£ A <s> u>/ G Xf; 



Proof. By the same argument as the proof of Lemma 4.3.4, we see that 

e (t) = (_i) l{(Cj)eA M- )|2t " l «' 2t( ^ )j ' Q «< J ' )= ~ 1}l 
Hence / G 9£ A if and only if w f G W(T)f and for all (£,j) G A^fe), 

^ = -l,2t(^) J ^2|TO ? ,2|(^) J , 
Clearly this is equivalent to w/ G AV V . 

As in §4.3, 

F 9 -rank(T)+F g -rank(Z G ((T n fKf- 1 ) )) 

Tl L 

= -Z + 4,fe)o +^0^2 mod 4 + rC/E)! mod 2. 



D 
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So Lusztig's formula gives (compare §4.2): 

(B,,Ind^(l)) = i(-l)f ]T (-l)^)°+^)--"+^(-) 12 ^) 



w£X- 



U_ e (-D 



Cfe)2mod4 + !C(ii)l 



Under the isomorphism FT^T)^ -^> J| f Z"< of §1.4, X^ v corresponds to 

Hence 

<B iy Ind^(l)> = i(-l)9 



/ 



n 

2fm 5 



E (-D 
y^=i^i, 5 (^)i=o 



C("«)+C 4 ;(^)2mod4+^(^)l 2 4 5 (>'<r) 



n 

? n <-> 



2|m 5 



E (- 



. 2) V«> 



2{mj 2|m £ 



"5 



In those factors of the second term for which mj is odd, we have used the fact that 
if w e eZ^_ inv , then 

C 5 (^)2.„od4 + \ll^h = ^T m od 2. 
So (|4.4.1|) follows by applying ( |4.3.2| ), ( |4.3.3| ), (|4. 1.2ft , (|4.1.3|) , and ( |2.1.2|) . 

5. Combinatorics of the Symmetric Group 

This section is devoted to the proof of the combinatorial facts invoked in §§2-4. 
The notation introduced in §1.2 will be used. We say that a function / on the set 
of partitions is multiplicative if 



f(v)=Hf(i m ' (u) ), vi/. 



Examples of multiplicative functions of v are e„, z v = l-Z^I, and |Z^ 1V |. 

Our first starting point is Q, VII. (2.4)], which as noted above is precisely 



(S) 



l^ff-invl 



Erf- 



p even 
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So to prove (4.1.3), it suffices to prove that 



e„ £ (-2)4W = |Z^_ inv |. 



To see this, note that since both sides are multiplicative, it suffices to consider the 
case when v is of the form (a 6 ), in which case both sides are 



0. 



6/2. 



6! 



2*>/2(!)!' 



if a is odd and b is odd, 

if a is odd and b is even, and 



yVM r (2r)! 

_> I „ ) a — , , if a is even. 



r=0 

This fact also implies 



(3.2.2) 



once we note that 



2rJ 2 r rl 



E ho* 00 = E * p - 

(w,c)e^_ inv pHH 

p even 



J2 (-1) £ -M = E (-1)^ M 2^M = e, E (~ 2 ) 4 



(") 



(w,e)eZ» 



U >£- Z inv 






since if there exists an involution w E Z v fixing no odd cycles of f, then ^(^)i must 
be even, so (— 1)^M = e u . 

Our second starting point is E3, 1.8 Example 11], which can be rewritten: 

So to prove ( |3.1.3 ), it suffices to show that 

i{ W g zr nv 1 4»o = too = on = E(- 1 ) £ " (I/) ' 

which we can again prove simply by observing that when v = (a b ) both sides are 



0. 



bji_ 



bl 



E 



2 b/2 (|)!' 
b\J2r)\ 



2rJ 2 r r\ 



if a is even and b is odd, 

if a is even and b is even, and 

, if a is odd. 



The remaining identities require a different approach. We first prove 



o 



E (- 1 ) " 5 = E iv^Cp'M- 



(»,E)e2, 



Cp"^"iP )- 



ph\u\ 
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By definition of induction product (|1J, 1.7]), and using ( [2.1. 2| ) 



E (-D 



tl(») 



(w,e)ezy, 

(p+,p )- 



min{p^ .p } 

E 



/ 



r=r p++p 2 ~- H i 



E 

{i\-\i'\-\-2r—p^~ —p 
pj even 

min{jp ,p~ } 

E E & ( p+ ~ r ' p ~ ~ r ' p')x£ 



^Mjv^-Ovlf 1 - r ) 



V 



K-) 



, — r p++p~-|"| -| pH|f | 



where, by Pieri's formula (H, (5.16)]), b{p + — r,p~ — r,p') is the number of ways 
of removing first a vertical (p~ — r)-strip, then a vertical (p + — r)-strip, from the 
Young diagram of p' , to leave a diagram with all rows of even length. Now every 
signed tableaux T £ T{p+ , p -){p') determines uniquely an r as above and such a way 
of removing strips, as follows: 

• order rows of equal length so that rows ending B are below rows ending EH ; 

• take the vertical (p~ — r)-strip to consist of all final boxes signed H; 

• take the vertical (p + — r)-strip to consist of all final boxes signed EH in rows 
of odd length, including those made odd by removal of the first strip. 



This correspondence is clearly bijective, which proves ( 3.1.2|) . 
The proof of 



(HI) 



£ (.^u^m. = £ (-lr^iv^prixs 



(w,e)£Z» _ 



ph\u\ 



proceeds similarly: 



V (_l)4(")0mod4+^Ml 



(w,e)eZ u , 




min{p ,p~ } 


= E 


(H 


r=r++P -_ k i 1 


V 


= E c (p + 'P^'P')x^ 


pHH 





|„| + 2 T ._p+_p- 



E 



xn-x {p r) -x ( 



fi\- 1 v I -\-2r — p + — f 
fi' even 



(w„) 
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where by the same bijection as before, 

i(|p|-|{rows of T ending B}| 
\ ^ / -i \ — | {odd rows of T ending EB}| — |{even rows of T ending B}|) 

= (— I)h(\p\-£(P'M \~^ f_ -n| {even rows of T ending B}| 



T * T (,+ ,p-)W 



Now 



(_l)f(|pMO')i) = (_n«(/)2mod4+^(p')3„od4 _ r^EsC^) = (— l) n 0>) 



and by grouping together signed tableaux whi ch diff er only in even rows it is easy 
to see that the sum equals \T/p+ iP -\(p')^\. So ( 3.3.2 ) is proved. 
Our next task is to modify this proof of ( 3.3.2 ) to derive 

V (-l)CM0mod4+|4Ml 

(i»,e)£Z,"_ inv 

( ^ } = e (-i) n(rt (n(^(p')+i))xe- 

p\-\v\ i 

2\m 2 i+i(p') 

We may assume that \v\ is even, for otherwise both sides vanish. Following the 
above pattern, we need to replace x^ p -X ^ with the class function on S\„\ _ 2r 
defined by 

w ^ \{(A+,A-) | {1, ■ •• , \v\ - 2r} = A+UA-, \A+\ = \A~\, u,(A±) = A*}\. 
It is easy to see that this is £ i i _ '"(-l) i X (M_2 ''' _M) . Thus 

V (_l)4(")0mod4+§C(«')l 



(io,e)£Z, 


'-inv 


m/ 


-E 


((-i) r E 


r=0 I ^H2r 
\ fl even 



\ 



>=() 



{w v ) 



J 



= E «%£, 

ph\u\ 

where, analogously to the above, we can write d(p') as the sum, over T 6 T( 00 )(p'), 
of a sign determined by T. As in the proof of ( |3.3.2| ), the (— l) r contribution to the 
sign is 

/ -i \n(p) + |{evcn rows of T ending H}| 

The (— I) 1 contribution is trickier to rephrase in terms of T, but an examination of 
the bijection 

Tab(p' - M ', (M M))_% ]J Tab°(p'-M / ,(M-2r--i,f)) 



i=0 
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defined in jlj, (9.4)] (in the proof of the Littlewood-Richardson Rule) reveals that 
the correct reformulation is 

(_1 \ I {rows of T ending B}|-m(T) 

where m(T) is the maximum, over all rows R of T, of the quantity 
| {odd rows ending B below or equal to R}\ 

— | {odd rows ending EH below or equal to R}\ 

(assuming that the rows of T are ordered, as above, so that rows ending B come 
below rows ending EE3 of the same length) . Hence 

dip') = ( — lW p ' V^ /-[\|{odd rows of T ending B}|-m(T) 

TeT (0 ,o) (/=>') 

and we are reduced to proving that 

V^ /i\|{odd rows of T ending B}|-m(T) 
TeT (0]0) (p') 

' Y[(m 2i ip') + 1), if2|m 24+ i(p'),Vt 

i 

0, otherwise. 

Grouping together signed tableaux which differ only in even rows, we see that we 
may assume that p' has only odd parts. In lieu of a direct proof, we can deduce 
this from ( 3.2.2| ), proved above. It says that 

E rt = E (-i) 4M 

p even 



E ( E **)•( E (-W" 1 

r=0 I fj,h2r i=0 , 

\ p' even / 



-2r—i,i) \ 



(Wv), 



whence 

1, if 2 | m; (//), V« 

0, otherwise. 



V* f 1 \|{rows of T ending B}|-m(T) _ I 

Ter (0i0) ( P ') *- 

When // has only odd parts, this is precisely the statement we want. 
It is now easy to prove (4.3.3), since 

(tu,£)6ZJ_ inv 



e v E (-l) ,l(p) (II( m ^(p') + l))x^ 



pHH 

2|m 2i+ i(p') 



E (- 1 ) n(p,) (II( m »^) +1 ))^. 



phii/i 

2|m 2 i+i(p) 
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as required. Finally, ([4.1.2 ) follows from ( 3.1. 2|) by summing over all signatures 



= (-l)Me tf E |T±(p')|x£ 

pHH 

= (-i) M E(II( m *0») + i))x5, 

pr-|i/| i 

and ( 4.3.2 ) follows from ( 3.3.2J ) in an analogous way: 



[3 

K 

p: 
p: 

[9 

[io: 

[H 
[12: 
[13; 

[14; 
[15; 

[16. 



V^-l^Wo+O^mocu+^Mi^i.C") = e„ V (-l)" (p) |T ± ( / 9') V 'lx^ 

w£LZy av pHH 

= E (-l)" (p,) (n( TO 2m(p) + l))xe- 

ph|i/| i 

2|m 2 ,(p) 
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